New ITG Guideline for the Usability Evaluation of Smart Home Environments

As a complement to the 2011 ITG Guideline 2.1-01 on the evaluation of terminal devices, a new ITG guideline on the usability of smart home environments has been drafted. This guideline addresses services and applications which allow for an implicit interaction between user and system taking place in an environment equipped with networked sensors and actuators, partially also without any dedicated user interface. The present paper reviews the current draft of the guideline by describing its scope, the underlying system set-up, the usability aspects and the evaluation criteria which might be considered relevant. 1 Motivation and Introduction In 2011, the ITG has agreed upon a guideline for the evaluation of terminal devices for communication applications [1]. This guideline provides an introduction to quality and usability as well as their underlying dimensions, classifies applications according to their function, usage scenario, and inputand output modalities, and further provides an introduction to evaluation methods which are useful in the product design cycle, following the usability engineering lifecycle [2]. The guideline than provides practical recommendations on how to select an appropriate evaluation method for a given purpose, how to design evaluation tasks, and how to carry out an evaluation is a proper way. Practical application examples of evaluations conclude the guideline. Whereas the 2011 guideline might provide helpful information for typical applications running on terminal devices (such as smartphone apps), it is limited when it comes to applications or services which are integrated into an environment, potentially equipped with sensors and actuators which are interconnected via an IT network. Such applications frequently allow for implicit interactions between user and systems, where the effect of a user action is perceived only via a change in the smart environment, and where system actions are provoked implicitly by sensors reacting to changes of the environment itself, or to user actions which are not explicitly directed to the system. Sometimes, services do not even provide a dedicated user interface which would allow for a proper evaluation, but are fully integrated into the environment. Proactive behavior of the service together with the absence of a user interface can then lead to the impression that the environment may be smart or intelligent. Recent developments lead to multimodal interaction concepts, often one modality enhanced with capabilities of speech. In order to cope with the limitations of the 2011 guideline, researchers and practitioners gathered around the department “services and applications” of the ITG (ITG FB 2) to start work on a new guideline related specifically to the evaluation of smart home environments. This guideline is conceptualized as a complement to the existing guideline, but should be formulated in a way which makes it understandable as a stand-alone document to practitioners, like designers and suppliers, who are the main intended addressees. The current draft of the guideline [3] is the outcome of a series of expert workshops which have taken place in 2013, as well as open calls for contributions launched through the ITG website. It has been submitted to the ITG FB 2 for further discussion, to be approved by the ITG later in 2014. It is the aim of the present paper to provide a synopsis of the rationale and the content of this new ITG guideline. Although the guideline does not specifically address speech as an interaction modality, many of its considerations also refer to speech. We will start with a definition of the scope of the guideline in Section 2. Section 3 provides a schematic view of the interaction between a user and a smart home system used throughout the guideline. Section 4 addresses different aspects of usability which are specifically connoted for smart home systems. Section 5 addresses criteria which can be used for an adequate evaluation. Section 6 concludes with a list of evaluation methods addressed by the guideline, as well as open questions conceivable for future work.